This program is developed based on the Shiny framework, a set of R packages and a collection of scripts written by members of Junhyong Kim Lab at University of Pennsylvania. Its goal is to facilitate fast and interactive RNA-Seq data analysis and visualization. Current version of PIVOT supports routine RNA-Seq data analysis including normalization, differential expression analysis, dimension reduction, correlation analysis, clustering and classification. Users can complete workflows of DESeq2, monocle and scde package with just a few button clicks. All analysis reports can be exported, and the program state can be saved, loaded and shared.
See http://kim.bio.upenn.edu/software/pivot.shtml for more details.
# dependecies that needs to be manually installed, you may need to paste the code line by line
install.packages("devtools")
library("devtools")
source("http://bioconductor.org/biocLite.R")
biocLite("GO.db")
biocLite("HSMMSingleCell")
biocLite("org.Mm.eg.db")
biocLite("org.Hs.eg.db")
biocLite("DESeq2")
# Install PIVOT
install_github("qinzhu/PIVOT")
install_github("qinzhu/PIVOT.launcher")
If you have 10x data output from Cell Ranger, please install Cell Ranger R Kit from https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/rkit to allow PIVOT to directly read in the data.
We are moving PIVOT to bioconductor for easier installation in the future, stay tuned.
To run PIVOT, in Rstudio console, use command
library(PIVOT)
pivot()For advanced users, if you want to only load needed modules,
Then you can either use
pivot_module()
which shows the available modules in PIVOT:
| ID | Module |
|---|---|
| 1 | PIVOT.analysis |
| 2 | DESeq2 |
| 3 | edgeR |
| 4 | scde |
| 5 | monocle |
| 6 | PIVOT.network |
| 7 | caret |
| 8 | PIVOT.toolkit |
Then use pivot(#ID_vector) to launch selected modules, e.g., pivot(c(1,2,3)) to launch PIVOT with the base PIVOT module, DESeq2 and edgeR.
Alternatively, use
pivot_launcher()
to launch a window to directly pick modules or install required components.
To input expression matrix, select “Counts Table” as input file type. PIVOT expects the count matrix to have rows as genes and samples as columns. Gene names and sample names should be the first column and the first row, respectively.
PIVOT support expression matrix in csv, txt, xls or xlsx formats. Choose proper settings on the left file input panel until the right “Loaded File Preview” correctly shows the data frame.
you need to make sure that the data matrix:
Contains no NA or non-numeric values.
Does not have duplicated feature or sample names (PIVOT will alert the user if it detects any).
You need to install Cell Ranger R Kit from 10x Genomics to allow PIVOT to directly read in 10x data: https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/rkit
Then choose “10x Directoy” for input file type, and use the “Select 10x Folder” button to point PIVOT to the Cell Ranger output directory (the folder containing the “outs” folder).
By default, PIVOT will use gene symbol as row names for the expression matrix, the other option is the gene id, which is the default setting of Cell Ranger output.
We have included a multitude of normalization methods in PIVOT. Some normalization methods are mostly used for bulk RNA-Seq data, while others may only be applicable to single-cell data. Users should be aware of which method best suits their data.
If your data has already been processed by DESeq or other methods, please specify “none” in the normalization method.
If DESeq failed on your data, one possibility is that you have low counts samples, which leads to all the genes contain at least one 0 in the counts matrix. You can either find out and remove these samples, or choose the “modified DESeq” normalization method. Setting sample inclusion 100% is equivalent to the original DESeq.
Some normalization method require additional information from the user. For example, if you choose ERCC normalization then you must provide experimental parameters. Please also make sure the ERCC feature names matches those in the standard table: (https://tools.thermofisher.com/content/sfs/manuals/cms_095046.txt).
PIVOT applies a pre-filtering step before doing normalization. By default, PIVOT will filter out genes with all 0 expressions. Users can also specify a different row mean or row sum threshold to remove those low confidence features.
Once data have been normalized, you can check the normlization details which contain information such as the estimated size factors.
The design infomation are used for sample point coloring and differential expression analysis. Users can input the entire sample meta sheet as design information, or manually specify groups or batches for each sample.
The first column of the design table should always be the sample name list, which must include all samples that’s in the expression matrix. The rest columns will be treated as “categories” or “design variables”, which can be “condition”, “batch”, “operator”, “experiment date”, etc. You will be able to choose which category to be used for analysis such as DE, as well as if the category should be treated as categorical or numerical.
You can also manually make a design-info file by specifying the sample grouping in PIVOT, and download it for later upload.
Empty or NA entries are allowed in design table. But You need to make sure the categories used for DE testing (conditions/time points/batches) do not contain NAs.
The design categories will be used as “meta” info for sample coloring in many modules, such as PCA or heatmap.
PIVOT supports automatic ID/Name conversion and supports most of the ID/Names listed in the BioMart database. Despite this powerful feature, we still recommend users to get the feature name right in the first place (i.e., use proper gene id/name output when doing the count quantification), as id/name conversion could inevitably lead to id or name duplicates, and unmapped features.
Expression Filter: PIVOT incorperates the scater package to provide users various QC plots for feature filtering. You can choose any of the stats and select range to only keep those features within the specified range.
Feature List Filter: You can provide a marker feature list to get a marker feature expression subset.
The marker features should appear as the first column in your file.
Some of the features in your marker list may not be found in the dataset, because they may have already been removed due to 0 expression in all your samples.
P-Value Filter:
Removing Filtering Effects
Subsetter allows you to choose a subset of samples for analysis. You can either manually select samples, groups, upload a sample list or subset based on sample statistics.
To filter based on sample statitics, you can directly drag on the sample stats plot to specify a range.
You can choose whether or not the subset should be renormalized. You can change the re-normalization method and associated parameters in the “FILE” panel.
An implicit filtering will occur to get nonzero count genes for the subset. This procedure prevents some downstream analysis from breaking on 0s.
Applying will return the data to the original input dataset, which means all filtering or subsetting effects will be removed.
With each data filtering/subsetting operation, you are creating data subsets whose lineage can be tracked using the data map. Mouse over each edge in the map will show you the operation details, and you can switch between data subsets, rename nodes, delete subsets or add notes by simply selecting the nodes and click buttons.
To attach analysis to the nodes, simply click the magnet button on the top right corner of each analysis box. The rest of the buttons are for pasteing R markdown reports to the report module, generating HTML reports and box collapsing.
For most analysis modules, you can choose one of the four data scales:
counts (normalized) : DESeq normalized counts;
Log10 : log10 (normalized_counts + 1). Plus one to include zeros;
Standardized : Standardization (calculate Z-scores) is performed on the DESeq normalized counts;
Log10 & Standardized : Standardization (calculate Z-scores) is performed on log10(normalized_counts + 1), assuming log-normal distribution.
For each individual analysis, please choose the most proper data scale. Some modules have fixed data scale choice (e.g., raw counts input for DESeq differential analysis) so this option is not available.
You can download data table with different data scales and ordering.
The relative frequency of a gene is defined as its raw count divided by the total counts of the sample.
Clicking features in the data table will plot its expression in the bottom panel, you can choose various plot types including point/bar plot, box plot or violin plot. You can also convert the plot to an interactive plot.
The sample statistics panel allow you to visualize various important sample statistics. For example, clicking a sample in the left table will plot the feature count distribution. Normlization method like Census have strong assumptions on the count distribution, so you can use this to check if your data matches their assumption.
You can also visualize important sample statistics as bar plot, histogram or density plots across samples.
Similarly, the feature statistics table can be clicked and the selected gene expression can be visualized.
Seurat package, a user can identify the variably expressed genes with any expression and dispersion cutoff, and download the gene list for input into other modules like heatmap.You can specify the full model formula and reduced model formula used for DESeq LRT test. The model formula must be constructed with valid design variables such as the input design categories, 0 or 1.
If use exact test:
Simply choose which pair to compare.
If use GLM model : If model contains intercept (formula ~0+XYZ), use contrast, e.g., contrast: (1)A (-1)B) for pairwise group comparison
If model does not contain intercept (default), the first group will be treated as the baseline. The coefficient B will be B vs A and coefficient C will be C vs A. To compare B vs C use contrast: (1)B (-1)C.
See edgeR manual for more detailed explanation.
Note currently there is a bug causing SCDE to fail in multiple occasions: https://github.com/hms-dbmi/scde/issues/48. PIVOT will update as soon as the SCDE team comes up with a fix.
The SCDE error modeling must be performed first before you can use other SCDE analysis. For large dataset the modeling process can be very slow. You can monitor the progress in the background R session.
You can use SCDE distance for hierarchical clustering and minimum-spanning-tree generation. There are three types of adjustment method you can choose: direct drop-out, reciprocal weighting and mode relative weighting. For details of these methods please check the SCDE website. Once a distance has been computed, it is loaded into PIVOT to be used in other modules.
PIVOT follows the monocle vignette (http://cole-trapnell-lab.github.io/monocle-release/docs/) and provide a graphical interface for the Monocle package.
To start, first initate monocle cellset object by choosing a proper distribution for your data, set a minimum expression level (corresponds to min_expr argument), and press “Create Monocle CellDataSet”.
You can use all genes (not recommended), variably expressed genes or monocle detected DEGs for cell state ordering. Once you select your ordering gene set, simply press “Set Ordering Genes”. You should be able to see the right side dispersion plot highlighting the selected ordering genes.
As described in the monocle documentation, it offers two anaylsis cell types: cell state ordering and unsupervised clustering. Once you choose the analysis type and specified the relavant parameters, press the run button to proceed.
If the analysis type is ordering, the result will add two columns to the sample meta sheet: Pseudotime and State. Clustering will generate a “Cluster” column.
If you have provided group information, PIVOT will automatically compare the assigned state/cluster to your groups by generating a confusion matrix and a comparison plot.
Cell state ordering will also generate a cell trajectory plot. You can click the gene list to view the expression level of the selected gene plotted on the graph or as a function of pseudotime or cell state. Note because the example dataset is composed of different cell types, the assigned state in the plot below does not have any biological meaning but only serve as a demo.
You can perform hierarchical clustering on various transformations of the expression matrix, as well as projection matrix of PCA, t-SNE, MDS or diffusion map. The latter requires you to have performed corresponding analysis first. For projections by PCA or diffusion map, you can further choose which sets of PC/DCs should be used as input for clustering.
You can color the leaves of the dendrogram by multiple sample meta data (design categories). You can use different color sets for different categories by specifying the same number of color palettes in the “group color” input box.
You can compare the clustering result to existing design categories using the confusion matrix.
This module is a graphical interface for the SC3 package. You can find the full manual for SC3 here: https://bioconductor.org/packages/release/bioc/vignettes/SC3/inst/doc/SC3.html.
It is recommended to estimate best K first to determine a range of ks for SC3 to explore. Clicking the button and you’ll see the k_estimation. Then you can proceed to run SC3. For large dataset we recommend using multiple cores.
The plot shows pairwise comparison between your samples. The x and y axis of each plot show log10 RPM estimates in the cell corresponding to a given column and row respectively. The set of smoothed scatter plots on the lower left shows the overall correspondence between the transcript abundances estimated in two given cells. The upper right corner shows three-component mixture model, separating genes that “drop-out” in one of the cells (green component shows drop/out events in the column cell, red component shows drop-out events in the row cell). The correlated component is shown in blue. The percent of genes within each component is shown in the legend.
This module is adapted from the SCDE package.
The sample correlation heatmap provides a more intuitive way of visualizing the correlation between your samples. If you specifies color by group, a color bar will be added to the heatmap to show the group info.
You can also adjust multiple aesthetics of the plot, and choose if the plot should be static or interactive by changing the plotting package.
You can rank features by fano factor, variance or mean expression. You are not allowed to use standardized data for ranking because all feature will have the same mean (0) and variance (1). However, if you want to plot in standardized scale, you can choose rank by “Row order”.
The ranking of the heatmap is used for choosing the top ranked features to be plotted in the heatmap. You can use the slider to choose the features to be plotted. By default if the number of features is greater than 500, only the top 500 features will be plotted. Using the manual input, you can choose any rank range other than the top 500. The only limit is that you can only plot 1000 features at a time.
alt text
By default, the order of the samples (columns) and features (rows) shown in the heatmap will be determined by hierarchical clustering. Alternatively, if “Do not cluster sample/feature” is specifed, the samples/features will be ordered by the rank. If the user also specifies “rank by row order”, then the heatmap order will be exactly the same as the input row/column order.
alt text
d3heatmap package allows zoom in (click and drag) and zoom out (double click).
Press to try different colors!
You can look at any principal component. Percentage shown on each axis is the percent variance explained by that principal component.
alt text
1D, 2D and 3D T-SNE are results of 3 different t-SNE processes (dims = 1, 2 or 3).
According to http://lvdmaaten.github.io/tsne/,
“Perplexity is a measure for information that is defined as 2 to the power of the Shannon entropy. The perplexity of a fair die with k sides is equal to k. In t-SNE, the perplexity may be viewed as a knob that sets the number of effective nearest neighbors. It is comparable with the number of nearest neighbors k that is employed in many manifold learners.”
alt text
This module is a graphical interface for the caret package (http://topepo.github.io/caret/index.html). Currently it allows the user to choose almost all classification models listed in ( http://topepo.github.io/caret/modelList.html).
Some methods may require new packages to be installed. In such cases, the background R session will ask you to install it. Choose yes if you want to proceed.
alt text
Users are expected to have the knowledge of which model is most suitable for their data. The details of these models can be found in the caret website.
Many methods contains built-in feature selection. For those having explicit coefficients for the features, the feature coefficient table will be available for download.
You can specify the cross-validation parameters to be used in the training.
alt text
For methods containing internal feature selection and explicit feature coefficients, the information will be available in the “Feature Coefficient” table.
The trained model can be used as a classifier for new data, or be used for the testing of the model. For example, if you train the classifier using data of group A and group B, and then perform testing on group A-t and group B-t. Then samples in group A-t and B-t will be classified as group A or group B, so that you can tell how well the model is performing. Alternatively, you can use the model to classify the unknow samples in e.g. group C.
The code has been pre-written for you – you only need to add it. Please always run the module before you add it to report, because PIVOT requires the module has a last state to capture all the parameters.
alt text
The report code is based on R-markdown v2 (http://rmarkdown.rstudio.com/). You can add comments, change titles or modify the code (not recommended) according to the syntax.
The system control menu is located at the top right corner. Users can use this panel to save the program state, launch new session or go to data management panels.
alt text
You can save the program state as an R data object. In this way you can make sure that you won’t lose your analysis progress, and you can share the state with others.
To load the saved state, go to File panel and choose PIVOT state in input file type. The session will auto refresh and immediately switch to the loaded state when the state uploading is complete.
alt text
Please note that although every analysis result will be kept in the state, PIVOT is not able to return the parameter choices to the ones you chose when the analysis was performed. In other words, the results and plots shown in the loaded state will stay the same, but the new parameter UI will return to the defaults.
Each time you exit, PIVOT will automatically save the state into the background R session. If you don’t close the R session, or save the workspace image before exiting R, the next time you launch PIVOT it will automatically load the state for you.
To clear the state, click .
Clicking the button will launch new PIVOT session for you. In this way you can do different things in different sessions. There is a limitation for this: because R is single-thread, you can only perform one analysis at a time. While R is busy computing in one session, the other sessions will have to wait in a queue.
If you really want to simultaneously perform multiple analysis in multiple sessions, currently the only way is to open multiple copies of R, and launch PIVOT using the command “pivot()” in each R session.
In all differential expression analysis modules, you can click the result table to view the expression plot of individual genes.
alt text
You can change the width of most plots by resizing the window.
You can resize ggvis plots by using the bottom right triangle, and download the plot by clicking the cog icon on top right.
alt text
The citation and licensing information can be found at the bottom of each module.
alt text
You can find in many places in PIVOT. You can click it to view the relavant information of the module.
Many parameter inputs has tooltips containing the relavant info.
alt text